In recent years, studies about mixed reality (to be abbreviated as MR hereinafter) that aims at seamless joint of real and virtual spaces have been extensively made. MR has received a lot of attention as a technique that aims at coexistence of a virtual space (to be referred to as VR (virtual reality) hereinafter) which can be experienced in only a situation separated from a physical space, and augments virtual reality.
An image display apparatus which presents MR is implemented by an apparatus which synthesizes an image of a virtual space (e.g., a virtual object, text information, and the like rendered by computer graphics (to be abbreviated as CG hereinafter)) onto an image of a physical space captured by an image sensing device such as a video camera or the like. For example, as an MR system, a system which additionally displays corresponding position information and a name (i.e., images of a virtual space) when the user wears an HMD (Head Mount Display) and looks at a landscape (i.e., an image of a physical space), or the like is known.
MR requires high-precision registration between the physical space and virtual space. Many efforts have been conventionally made for such registration problem.
The registration problem in MR amounts to obtaining the three-dimensional position and orientation of a camera as a viewpoint position on a coordinate system (to be referred to as a world coordinate system hereinafter) set on the physical space. Note that the world coordinate system need not be fixed on the physical space. For example, if the interior of a vehicle is defined as a virtual space, the world coordinate system set on the physical space can be set to move in accordance with the movement of the vehicle.
As a method of measuring the position and orientation of the camera on the world coordinate system, a three-dimensional (3 D) position/orientation sensor such as a magnetic sensor, optical sensor, ultrasonic sensor, or the like may be used. However, since the camera position measured by such position/orientation sensor has insufficient precision, if a virtual object is synthesized and displayed on the physical space on the basis of the position and orientation measured by the sensor, the object is displayed at a position deviated from a required position.
To solve this problem, conventionally, as a method of correcting sensor errors or a method of measuring the position and orientation of the camera on the world coordinate system without using any sensor, the following method is available (see, e.g., Japanese Patent Laid-Open No. 2002-228442 (U.S. Pat. No. 6,834,250)). That is, a plurality of indices (markers) whose coordinate (world coordinate) values on a 3 D space are known are laid out and are sensed by the camera, and the position and orientation of the camera that meet the relationship between the world coordinates of markers and sense image coordinates are calculated.
In general, if the positions of a plurality of points (three or more points theoretically; six or more points required for a stable solution) whose 3 D positions are known on the sense image can be obtained, the position and orientation of the camera viewpoint can be calculated based on their correspondence (to solve a so-called PnP problem).
When such registration method using markers is used, markers whose world coordinate values are known must be prepared in advance. An operation required to prepare for markers whose world coordinate values are known for the purpose of registration will be referred to as “marker registration” hereinafter. The markers may be one which are artificially laid out or may use natural features or the like. The markers are preferably laid out to always be sensed by a moving camera.
Marker registration is to input information such as features, locations, and the like of markers by laying out markers on the physical space (or for the markers that have already been laid out). In marker registration, the operation for precisely inputting 3 D layout information of the markers imposes a heavy load on the operator, and it is difficult to lay out the markers so as to always be sensed by the moving camera.
In the conventional system which makes the user who wears the HMD experience MR, there is no means for readily adding markers. For example, if the user feels the need for laying out new markers since no markers appear on an HMD camera image due to an insufficient number of markers which are laid out, he or she cannot easily and additionally register markers while wearing the HMD.